Gain Variation in Recurrent Error Propagation Networks

Abstra ct. A global gai n term is int rod uced into recurr ent a na log network s. T his gain ter m may he va ried as a. recurrent netwo rk se t-tles, similar to the way temperatu re is varied when "a nnealing" a networ k of stoc hast ic binary unit s. An err or propag ation algor it hm is presented which simultaneou sly opt imizes the weights and the gain schedule for a. recurrent network. Th e performance of thi s algorithm is comp ar ed to the sta nda rd back propagation algorit hm on a. diffi cult const raint sa tisfaction problem. An orde r of magnitude imp rovement in t he numb er of learni ng t rials requir ed is observed with t he new algorithm. Thi s improvement is obtained by allowing a. much larger region of weight space to sat isfy the problem. T he simultaneous opti mizat ion of weights and gain schedule lead s to a qualitatively different region of weight space t han t hat reached by op tim izati on of the weights alone. 1. I nt r o d u ct io n Neural networks ha ve received m uch att ent ion recentl y as plausible models for st udy ing the computational propert ies of massively parallel systems. Th ey have been app lied to a variety of problem s in signal processing, pattern recogniti on, speech processing, and more t raditional AI pro blem s. T hese models developed from ea rlier work in associative memo ries a nd models of cooperat ive compu tation in early vision processing {3,5,19]. Learni ng algori thms have been developed [1 ,18] that ena ble th ese networks to learn internal rep resentatio ns, allowing t hem to represent complex non-linear ma ppings. Two dist inct typ es of networks have been st udied qu ite exte nsively. T he first of t hese uses analog un its wit h a sigmoida l I/ O funct ion [8], and an erro r propagation algorithm for upd at ing t he weights to minim ize an error funct ion [16,17]. 1,,105t of t hese st udies have focused on st rict ly feed-forward networks. T he second type of network em ploys sto chast ic bina ry "T he a ut hor is curr …